Altmetrics and Grey Literature: Perspectives and Challenges

Traditional metrics largely overlook grey literature. The new altmetrics introduced in 2010 as “new, online scholarly tools (that allow) to make new filters” (Altmetrics Manifesto), can include all kinds of scholarly output which makes them interesting for grey literature. The topic of our paper is the connection between altmetrics and grey literature. Do altmetrics offer new opportunities for the development and impact of grey literature? In particular, the paper explores how altmetrics could add value to grey literature, in particular how reference managers, repositories, academic search engines and social networks can produce altmetrics of dissertations, reports, conference papers etc. We explore, too, how new altmetric tools incorporate grey literature as source for impact assessment, and if they do. The discussion analyses the potential but also the limits of the actual application of altmetrics to grey literatures and highlights the importance of unique identifiers, above all the DOI. For the moment, grey literature missed the opportunity to get on board of the new movement. However, getting grey literature into the heart of the coming mainstream adoption of altmetrics is not only essential for the future of grey literature in open science but also for academic and institutional control of research output and societal impact. This can be a special mission for academic librarians. Introduction Traditional metrics largely overlook grey literature. Worse, they basically disregard grey literature as irrelevant for the evaluation of research. Established metrics for individuals and organisations are journal-centric. Measuring the performance and popularity of scientists or research structures means counting the number of articles citing other articles, resulting in journal impact factors, normalized citation rates and the h-index. Even those rare studies including conference papers are limited to published proceedings. Grey literature remains out of scope. The most important reason is the way these metrics are produced – they rely on bibliographic tools like the Web of Sciences (WoS) and Scopus which from the beginning on were (nearly) exclusively journal and monograph A&I services, dismissing other vectors of scientific communication outside of the academic publishing market. The emergence of webometrics, i.e. the “study of the quantitative aspects of the construction and use of information resources, structures and technologies on the web drawing on bibliometric and informetric approaches” (Björneborn & Ingwersen 2004, p. 1217), change the situation. As many scholarly activities today are web-based, the field of webometrics is partially covered by scientometrics (figure 1). These new or alternative metrics are not limited to journals but apply to academic content (scholarly work) at large, insofar and as long as this content is available on the web, in particular on the social web (Galligan & Dyas-Correia 2013). They are sometimes called scholarly metrics or social media metrics, and most often defined as altmetrics. 1 See for instance Ingwersen et al. 2014, also for similar, older studies 2 The methodological problems to identify theses in bibliographic databases in Larivière et al. (2008) confirm the situation Schöpfel & Prost, Altmetrics and Grey Literature 2 The fact that these new metrics can include all kinds of scholarly output makes them interesting for grey literature. In a draft on altmetrics definitions and use cases, the National Information Standards Organization describes scholarly output as “a product created or executed by scholars and investigators in the course of their academic and/or research efforts. Scholarly output may include but is not limited to journal articles, conference proceedings, books and book chapters, reports, theses and dissertations, edited volumes, working papers, scholarly editions, oral presentations, performances, artifacts, exhibitions, online events, software and multimedia, composition, designs, online publications, and other forms of intellectual property” (NISO 2016, p.9). One part of this output clearly belongs to grey literature, especially when citable and accessible. Figure 1: Webometrics in the field of library and information sciences (source: Björneborn & Ingwersen 2004) The topic of our paper is the connection between altmetrics and conference proceedings, reports, theses and dissertations, and working papers. Do altmetrics offer new opportunities for the development and impact of grey literature? Are there already examples of good practice? Are there any barriers? However, before we outline the potential of altmetrics for grey literature, we will provide some elements for a better understanding of this concept. A short history of altmetrics Altmetrics have a short history. The term was introduced by Jason Priem from Chapel Hill in 2010, in a tweet published on the 29 September 2010: “I like the term #articlelevelmetrics, but it fails to imply *diversity* of measures. Lately, I'm liking #altmetrics”. This came after the global success of the web 2.0 tools and media, such as Facebook, Twitter etc., and it became popular as a kind of marketing umbrella for a broad range of new metrics of scholarly impact on the social web (Priem & Hemminger 2010). 3 See the definition of “acceptable products” by the National Science Foundation, Grant Proposal Guide II-12 NSF 14-1, November 2013 http://www.nsf.gov/pubs/policydocs/pappguide/nsf14001/gpgprint.pdf 4 See comprehensive reviews by Erdt et al. (2016) and Sugimoto et al. (2016) 5 https://twitter.com/jasonpriem/status/25844968813 by @jasonpriem Schöpfel & Prost, Altmetrics and Grey Literature 3 The Altmetrics Manifesto from 26 October 2010 merges article-level metrics and distributed scientific evaluation with social media into research on altmetrics and defines them as fast and open filters to relevant and significant scholarly sources, not in continuity but in disruption with webometrics or citations; “given the crisis facing existing filters and the rapid evolution of scholarly communication, the speed, richness, and breadth of altmetrics make them worth investing in” (Priem et al. 2010). From that moment on, the interest for altmetrics increased steadily to join and finally exceed scientometrics, according to Google Trends (figure 2). Two years after the Manifesto, the San Francisco Declaration on Research Assessment (DORA), initiated by the American Society for Cell Biology (ASCB), recognizes the need to improve the ways in which the outputs of scientific research are evaluated and suggests the “use of a range of article metrics and indicators on personal/supporting statements, as evidence of the impact of individual published articles and other research” (DORA 2012). Signed by nearly 12,500 individuals and 800+ organizations, DORA fostered the awareness for altmetrics and became a reference for the debate, research and development in the field. Figure 2: Altmetrics (blue) on Google Trends, compared to scientometrics (red) (2010-2016) The increasing number of scholarly work dedicated to altmetrics reveals the same trend (figure 3). No study on altmetrics before 2010, and then a steadily growth from 8 references in 2010 to 122 in 2015. 6 http://altmetrics.org/manifesto/ 7 http://www.ascb.org/dora/ accessed 7 September 2016 8 Data source: Google Trends www.google.com/trends accessed Sept 3, 2016 Schöpfel & Prost, Altmetrics and Grey Literature 4 The Google Scholar statistics confirm the Google Trend figures – the interest for scientometrics remains relatively stable, with 50-70 publications per year, but is exceeded by works on altmetrics from 2013 on. Figure 3: Publications on altmetrics and scientometrics Basically, altmetrics are “social web metrics for academic publications” (Sud & Thelwall 2014, p.1131) and particularly interesting for measuring societal impact, beyond the academic community (Piwowar 2013), through the count of views, downloads, clicks, likes, tags, posts (blogging) and tweets (micro-blogging), shares, discussions etc. The term “usually describes metrics that are alternative to the established citation counts and usage stats—and/or metrics about alternative research outputs, as opposed to journal articles” (NISO 2014, p.4). Variety is one main feature of altmetrics, a class of indicators measuring attention, dissemination and influence, even if the distinction between attention, dissemination and influence is not self-evident. The main areas of altmetrics are shown in figure 4. Impact on the (social) web can be assessed through the count of PDF or HTML downloads (viewed), the creation of references in online reference managers like CiteULike, Zotero or Mendeley (saved), the number of posts in blogs and micro-blogs, on Facebook or Wikipedia (discussed), the number of mentions in editorials or tools like F1000 (recommended) or as usual, simply via the number of citations in the WoS, Scopus, PubMed Central or CrossRef (cited). 9 Data source: Google Scholar https://scholar.google.fr allintitle: altmetrics (or scientometrics), accessed Sept 5, 2016 10 See https://www.altmetric.com/about-altmetrics/what-are-altmetrics/ Schöpfel & Prost, Altmetrics and Grey Literature 5 The NISO Alternative Assessment Metrics Initiative (2016) defines altmetrics as a broad concept that includes “multiple forms of assessment that are derived from activity and engagement among diverse stakeholders and scholarly outputs in the research ecosystem”. Today, a clear, common, widely accepted definition is not in sight. Altmetrics comprise many different types of metrics in a constantly changing landscape and “refer to a heterogeneous subset of scholarly metrics and are a proper subset of informetrics, scientometrics and webometrics” (Haustein 2016, p.416). Perhaps a pragmatic approach like Altmetric’s recent definition will fit best, for the moment: “Altmetrics are attention data from the social web that can help librarians understand which articles, journals, books, datasets, or other scholarly outputs are being discussed, shared, recommended, saved, or otherwise used online. They can be reported at the item-, journal-, or author-level”. Figure 4: Altmetrics areas of assessment Six years after the Manifesto, however, it is not quite clear if altmetrics are “an alternative or enhancement to the use of journal impact factors and click-through rate analysis to measure the impact and value of scholarly work” (Galligan & Dyas-Correia 2013, p.56). But they are already relevant for research evaluation. The European Commission DG Research and Innovation has established an Expert Group on Altmetrics which describes the emergence of altmetrics as part of the “transition to a more accountable and transparent research system”, more efficient, open to society, and expects “robust, responsible, transparent and interoperable uses of metrics and altmetrics in open science”. Altmetrics are levers in support of open science. Up to now, including altmetrics in decisions on grants, hiring and tenure still requires careful consideration but they may soon become a normal part of a CV (Kwok 2013). What does this mean for grey literature? What is the potential of altmetrics for grey literature? The next section tries to provide a global answer. 11 https://www.altmetric.com/blog/altmetrics-collection-development/ 12 Next-generation altmetrics: responsible metrics and evaluation for open science, available at https://ec.europa.eu/research/openscience/index.cfm?pg=altmetrics_eg cite view